Skip to contentSkip to conversation

mmtests-compare and minor fix

PATCH In review Luis Chamberlain 2 participants 5 messages Aug 12, 2025 – Aug 14, 2025

lore
Share & tools
lore
#0 lore raw

cover mmtests-compare and minor fix

While running mmtests I realized the html graphs were not being generaed. So kick that off and also try to enhance graphing. We can fine tune this later with time. The goal is to easily be able to do AB testing with mmtests against different kernels.

Also a small fix for gen_hosts is provided due to a patch from Daniel which forgot to update the kdevops_hosts in that file.

Luis Chamberlain (2):
gen_hosts: fix inventory path for dedicated mmtests workflow
mmtests: add AB testing and comparison support

 .gitignore                                    |   2 +
 defconfigs/mmtests-ab-testing                 |  22 +
 defconfigs/mmtests-ab-testing-thpcompact      |  31 +
 playbooks/mmtests-compare.yml                 |   5 +
 playbooks/mmtests.yml                         |   2 +-
 playbooks/roles/gen_hosts/tasks/main.yml      |   2 +-
 .../roles/gen_hosts/templates/mmtests.j2      |  10 +
 playbooks/roles/mmtests/tasks/main.yaml       |  58 +-
 .../roles/mmtests_compare/defaults/main.yml   |   6 +
 .../mmtests_compare/files/apply_patch.sh      |  32 +
 .../files/embed_graphs_in_html.sh             | 100 +++
 .../mmtests_compare/files/generate_graphs.sh  | 148 +++++
 .../files/generate_html_with_graphs.sh        |  89 +++
 .../mmtests_compare/files/run_comparison.sh   |  58 ++
 .../roles/mmtests_compare/tasks/main.yml      | 472 ++++++++++++++
 .../templates/comparison_report.html.j2       | 414 ++++++++++++
 scripts/generate_mmtests_graphs.py            | 598 ++++++++++++++++++
 workflows/mmtests/Makefile                    |  19 +-
 ...fined-array-reference-when-no-operat.patch |  46 ++
 ...act-fix-library-order-in-gcc-command.patch |  33 +
 20 files changed, 2139 insertions(+), 8 deletions(-)
 create mode 100644 defconfigs/mmtests-ab-testing
 create mode 100644 defconfigs/mmtests-ab-testing-thpcompact
 create mode 100644 playbooks/mmtests-compare.yml
 create mode 100644 playbooks/roles/mmtests_compare/defaults/main.yml
 create mode 100755 playbooks/roles/mmtests_compare/files/apply_patch.sh
 create mode 100755 playbooks/roles/mmtests_compare/files/embed_graphs_in_html.sh
 create mode 100755 playbooks/roles/mmtests_compare/files/generate_graphs.sh
 create mode 100755 playbooks/roles/mmtests_compare/files/generate_html_with_graphs.sh
 create mode 100755 playbooks/roles/mmtests_compare/files/run_comparison.sh
 create mode 100644 playbooks/roles/mmtests_compare/tasks/main.yml
 create mode 100644 playbooks/roles/mmtests_compare/templates/comparison_report.html.j2
 create mode 100644 scripts/generate_mmtests_graphs.py
 create mode 100644 workflows/mmtests/fixes/0001-compare-Fix-undefined-array-reference-when-no-operat.patch
 create mode 100644 workflows/mmtests/fixes/0002-thpcompact-fix-library-order-in-gcc-command.patch
Signature · 1 line
2.47.2
#1 lore raw

PATCH 1/2 gen_hosts: fix inventory path for dedicated mmtests workflow

The dedicated mmtests workflow generation was using kdevops_hosts variable instead of ansible_cfg_inventory, causing the generated hosts file to be placed in the wrong location. This resulted in ansible not finding the correct inventory for mmtests runs.

Use ansible_cfg_inventory which correctly points to the inventory path configured in ansible.cfg, ensuring the generated hosts file is placed where ansible expects it.

Fixes 0987c30034c995 ("gen_hosts: add initial support for dedicated mmtests workflow")
Generated-by Claude AI
---
 playbooks/roles/gen_hosts/tasks/main.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
playbooks/roles/gen_hosts/tasks/main.yml +1 -1
index 15c1ddbb..4f29ce30 100644
@@ -342,7 +342,7 @@
   tags: [ 'hosts' ]
   template:
     src: "{{ kdevops_hosts_template }}"
-    dest: "{{ topdir_path }}/{{ kdevops_hosts }}"
+    dest: "{{ ansible_cfg_inventory }}"
     force: yes
     trim_blocks: True
     lstrip_blocks: True
Signature · 1 line
2.47.2
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] gen_hosts: fix inventory path for dedicated mmtests workflow

Quoted 10 lines · Luis Chamberlain

The dedicated mmtests workflow generation was using kdevops_hosts variable instead of ansible_cfg_inventory, causing the generated hosts file to be placed in the wrong location. This resulted in ansible not finding the correct inventory for mmtests runs.

Use ansible_cfg_inventory which correctly points to the inventory path configured in ansible.cfg, ensuring the generated hosts file is placed where ansible expects it.

Fixes 0987c30034c995 ("gen_hosts: add initial support for dedicated mmtests workflow")

The commit header description is wrong. It should be:

Fixes 0987c30034c9 ("Makefile: use inventory from ansible.cfg")
Quoted 11 lines
Generated-by Claude AI
---
 playbooks/roles/gen_hosts/tasks/main.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
playbooks/roles/gen_hosts/tasks/main.yml +0 -0
index 15c1ddbb..4f29ce30 100644

playbooks/roles/gen_hosts/tasks/main.yml line 348

Quoted 8 lines
playbooks/roles/gen_hosts/tasks/main.yml +1 -1
@@ -342,7 +342,7 @@
   tags: [ 'hosts' ]
   template:
     src: "{{ kdevops_hosts_template }}"
-    dest: "{{ topdir_path }}/{{ kdevops_hosts }}"
+    dest: "{{ ansible_cfg_inventory }}"
     force: yes
     trim_blocks: True
     lstrip_blocks: True

This is incomplete.

Can you also add support for localhost to the mmtests hosts template as in commit 1cf0800c9ffc ("gen_hosts: templates: include localhost in the all group")? And update mmtests hosts field from all to baseline:dev?

Sorry I forgot about these changes. I can see I did only update the makefile when rebasing and I forgot about these 2 changes.

With that,

Thanks!

#3 lore raw

Re: [PATCH 1/2] gen_hosts: fix inventory path for dedicated mmtests workflow

Quoted 10 lines · Daniel Gomez
Quoted 9 lines · Luis Chamberlain

The dedicated mmtests workflow generation was using kdevops_hosts variable instead of ansible_cfg_inventory, causing the generated hosts file to be placed in the wrong location. This resulted in ansible not finding the correct inventory for mmtests runs.

Use ansible_cfg_inventory which correctly points to the inventory path configured in ansible.cfg, ensuring the generated hosts file is placed where ansible expects it.

Quoted 5 lines
Quoted 1 line
Fixes 0987c30034c995 ("gen_hosts: add initial support for dedicated mmtests workflow")

The commit header description is wrong. It should be:

Fixes 0987c30034c9 ("Makefile: use inventory from ansible.cfg")

Funny!

Quoted 3 lines

With that,

Pushed!

Luis

#4 lore raw

PATCH 2/2 mmtests: add AB testing and comparison support

In reply to Luis Chamberlain

This commit introduces comprehensive A/B testing and comparison capabilities for the mmtests workflow in kdevops, enabling performance regression detection between baseline and development kernels.

Key Features Added:
- A/B testing configuration support for mmtests workflow
- Automated performance comparison between baseline and dev nodes
- Visual performance analysis with graph generation
- HTML reports with embedded performance graphs

New Components:
1. Defconfigs:
- mmtests-ab-testing: Basic A/B testing setup
- mmtests-ab-testing-thpcompact: Advanced config with monitoring

2. Comparison Infrastructure (playbooks/roles/mmtests_compare/):
- Automated result collection from baseline and dev nodes
- Local mmtests repository management with patch support
- Multiple comparison output formats (HTML, text, graphs)
- Shell scripts for graph generation and HTML embedding

3. Playbook Integration:
- mmtests-compare.yml: Orchestrates the comparison workflow
- Updated mmtests.yml to target mmtests group specifically
- Enhanced hosts template with localhost and mmtests group

4. Result Visualization:
- Performance graphs (main, sorted, smoothed trends)
- Monitor data visualization (vmstat, mpstat, proc stats)
- Professional HTML reports with embedded graphs
- Comprehensive comparison tables with statistical analysis

5. Workflow Enhancements:
- Support for applying patches from fixes directory
- Python script for advanced graph generation
- Makefile targets for comparison workflow
- Results organization in workflows/mmtests/results/

Technical Improvements:
- Added localhost to mmtests hosts template for local operations
- Added dedicated mmtests group definition in hosts template
- Support for applying patches from fixes directory
- Robust error handling in comparison scripts
- Dependency management for Perl and Python tools
- Temporary file management in /tmp for comparisons

Included Patches:
- Fix undefined array reference in mmtests compare
- Fix library order in thpcompact gcc command

The implementation supports the standard kdevops A/B testing pattern where baseline nodes run the stable kernel and dev nodes run the development kernel, with automated comparison and visualization of performance differences between them.

Usage:
make defconfig-mmtests-ab-testing
make bringup
make mmtests
make mmtests-compare

This enables developers to quickly identify performance regressions and improvements between kernel versions with professional-quality reports and visualizations.

Generated-by Claude AI
---
 .gitignore                                    |   2 +
 defconfigs/mmtests-ab-testing                 |  22 +
 defconfigs/mmtests-ab-testing-thpcompact      |  31 +
 playbooks/mmtests-compare.yml                 |   5 +
 playbooks/mmtests.yml                         |   2 +-
 .../roles/gen_hosts/templates/mmtests.j2      |  10 +
 playbooks/roles/mmtests/tasks/main.yaml       |  58 +-
 .../roles/mmtests_compare/defaults/main.yml   |   6 +
 .../mmtests_compare/files/apply_patch.sh      |  32 +
 .../files/embed_graphs_in_html.sh             | 100 +++
 .../mmtests_compare/files/generate_graphs.sh  | 148 +++++
 .../files/generate_html_with_graphs.sh        |  89 +++
 .../mmtests_compare/files/run_comparison.sh   |  58 ++
 .../roles/mmtests_compare/tasks/main.yml      | 472 ++++++++++++++
 .../templates/comparison_report.html.j2       | 414 ++++++++++++
 scripts/generate_mmtests_graphs.py            | 598 ++++++++++++++++++
 workflows/mmtests/Makefile                    |  19 +-
 ...fined-array-reference-when-no-operat.patch |  46 ++
 ...act-fix-library-order-in-gcc-command.patch |  33 +
 19 files changed, 2138 insertions(+), 7 deletions(-)
 create mode 100644 defconfigs/mmtests-ab-testing
 create mode 100644 defconfigs/mmtests-ab-testing-thpcompact
 create mode 100644 playbooks/mmtests-compare.yml
 create mode 100644 playbooks/roles/mmtests_compare/defaults/main.yml
 create mode 100755 playbooks/roles/mmtests_compare/files/apply_patch.sh
 create mode 100755 playbooks/roles/mmtests_compare/files/embed_graphs_in_html.sh
 create mode 100755 playbooks/roles/mmtests_compare/files/generate_graphs.sh
 create mode 100755 playbooks/roles/mmtests_compare/files/generate_html_with_graphs.sh
 create mode 100755 playbooks/roles/mmtests_compare/files/run_comparison.sh
 create mode 100644 playbooks/roles/mmtests_compare/tasks/main.yml
 create mode 100644 playbooks/roles/mmtests_compare/templates/comparison_report.html.j2
 create mode 100644 scripts/generate_mmtests_graphs.py
 create mode 100644 workflows/mmtests/fixes/0001-compare-Fix-undefined-array-reference-when-no-operat.patch
 create mode 100644 workflows/mmtests/fixes/0002-thpcompact-fix-library-order-in-gcc-command.patch
.gitignore +2 -0
index 2e28c3f7..095880ab 100644
@@ -67,6 +67,8 @@ workflows/ltp/results/
 workflows/nfstest/results/
 
 workflows/sysbench/results/
+workflows/mmtests/results/
+tmp
 
 playbooks/roles/linux-mirror/linux-mirror-systemd/*.service
 playbooks/roles/linux-mirror/linux-mirror-systemd/*.timer
defconfigs/mmtests-ab-testing new file +22 -0
new file mode 100644
index 00000000..5d4dd2db
@@ -0,0 +1,22 @@
+CONFIG_GUESTFS=y
+CONFIG_LIBVIRT=y
+
+CONFIG_WORKFLOWS=y
+CONFIG_WORKFLOW_LINUX_CUSTOM=y
+
+CONFIG_BOOTLINUX=y
+
+# Enable baseline and dev testing
+CONFIG_KDEVOPS_BASELINE_AND_DEV=y
+
+# Enable mmtests workflow
+CONFIG_WORKFLOWS_TESTS=y
+CONFIG_WORKFLOWS_LINUX_TESTS=y
+CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
+CONFIG_KDEVOPS_WORKFLOW_DEDICATE_MMTESTS=y
+
+# mmtests configuration - using defaults
+CONFIG_MMTESTS_ENABLE_THPCOMPACT=y
+
+# Filesystem for tests
+CONFIG_MMTESTS_FS_XFS=y
defconfigs/mmtests-ab-testing-thpcompact new file +31 -0
new file mode 100644
index 00000000..cbcb30b9
@@ -0,0 +1,31 @@
+CONFIG_GUESTFS=y
+CONFIG_LIBVIRT=y
+
+CONFIG_WORKFLOWS=y
+CONFIG_WORKFLOW_LINUX_CUSTOM=y
+
+CONFIG_BOOTLINUX=y
+
+# Enable baseline and dev testing
+CONFIG_KDEVOPS_BASELINE_AND_DEV=y
+
+# Enable A/B testing with different kernel references
+CONFIG_BOOTLINUX_AB_DIFFERENT_REF=y
+
+# Enable mmtests workflow
+CONFIG_WORKFLOWS_TESTS=y
+CONFIG_WORKFLOWS_LINUX_TESTS=y
+CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
+CONFIG_KDEVOPS_WORKFLOW_DEDICATE_MMTESTS=y
+
+# mmtests configuration
+CONFIG_MMTESTS_ENABLE_THPCOMPACT=y
+CONFIG_MMTESTS_ITERATIONS=5
+CONFIG_MMTESTS_MONITOR_INTERVAL=1
+CONFIG_MMTESTS_MONITOR_ENABLE_FTRACE=y
+CONFIG_MMTESTS_MONITOR_ENABLE_PROC_MONITORING=y
+CONFIG_MMTESTS_MONITOR_ENABLE_MPSTAT=y
+CONFIG_MMTESTS_PRETEST_THP_SETTING="always"
+
+# Filesystem for tests
+CONFIG_MMTESTS_FS_XFS=y
playbooks/mmtests-compare.yml new file +5 -0
new file mode 100644
index 00000000..7e948672
@@ -0,0 +1,5 @@
+---
+- hosts: localhost
+  roles:
+    - role: mmtests_compare
+      when: kdevops_baseline_and_dev|bool
playbooks/mmtests.yml +1 -1
index f66e65db..4e395db6 100644
@@ -1,4 +1,4 @@
 ---
-- hosts: all
+- hosts: mmtests
   roles:
     - role: mmtests
playbooks/roles/gen_hosts/templates/mmtests.j2 +10 -0
index d32ffe40..1252fe87 100644
@@ -1,4 +1,5 @@
 [all]
+localhost ansible_connection=local
 {% for test_type in mmtests_enabled_test_types %}
 {{ kdevops_host_prefix }}-{{ test_type }}
 {% if kdevops_baseline_and_dev %}
@@ -21,3 +22,12 @@ ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 {% endif %}
 [dev:vars]
 ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
+[mmtests]
+{% for test_type in mmtests_enabled_test_types %}
+{{ kdevops_host_prefix }}-{{ test_type }}
+{% if kdevops_baseline_and_dev %}
+{{ kdevops_host_prefix }}-{{ test_type }}-dev
+{% endif %}
+{% endfor %}
+[mmtests:vars]
+ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
playbooks/roles/mmtests/tasks/main.yaml +57 -1

70 lines not on this page · raw message

playbooks/roles/mmtests_compare/defaults/main.yml new file +6 -0
new file mode 100644
index 00000000..201a5278
@@ -0,0 +1,6 @@
+---
+# mmtests compare role defaults
+mmtests_data_dir: "{{ data_path }}/mmtests"
+mmtests_results_dir: "{{ mmtests_data_dir }}/work/log/{{ inventory_hostname }}-{{ kernel_version.stdout }}"
+# Git URL is from extra_vars.yaml, fallback to GitHub
+mmtests_git_url: "{{ mmtests_git_url | default('https://github.com/gormanm/mmtests.git') }}"
playbooks/roles/mmtests_compare/files/apply_patch.sh new file +32 -0
new file mode 100755
index 00000000..172a5bd1
@@ -0,0 +1,32 @@
+#!/bin/bash
+# Script to apply mmtests patches with proper error handling
+
+TOPDIR="$1"
+PATCH_FILE="$2"
+
+cd "$TOPDIR/tmp/mmtests" || exit 1
+
+PATCH_NAME=$(basename "$PATCH_FILE")
+
+# Check if patch is already applied by looking for the specific fix
+if grep -q "if (@operations > 0 && exists" bin/lib/MMTests/Compare.pm 2>/dev/null; then
+    echo "Patch $PATCH_NAME appears to be already applied"
+    exit 0
+fi
+
+# Try to apply with git apply first
+if git apply --check "$PATCH_FILE" 2>/dev/null; then
+    git apply "$PATCH_FILE"
+    echo "Applied patch with git: $PATCH_NAME"
+    exit 0
+fi
+
+# Try with patch command as fallback
+if patch -p1 --dry-run < "$PATCH_FILE" >/dev/null 2>&1; then
+    patch -p1 < "$PATCH_FILE"
+    echo "Applied patch with patch command: $PATCH_NAME"
+    exit 0
+fi
+
+echo "Failed to apply $PATCH_NAME - may already be applied or conflicting"
+exit 0  # Don't fail the playbook
playbooks/roles/mmtests_compare/files/embed_graphs_in_html.sh new file +100 -0

100 lines not on this page · raw message

playbooks/roles/mmtests_compare/files/generate_graphs.sh new file +148 -0

148 lines not on this page · raw message

playbooks/roles/mmtests_compare/files/generate_html_with_graphs.sh new file +89 -0

89 lines not on this page · raw message

playbooks/roles/mmtests_compare/files/run_comparison.sh new file +58 -0

58 lines not on this page · raw message

playbooks/roles/mmtests_compare/tasks/main.yml new file +472 -0

472 lines not on this page · raw message

playbooks/roles/mmtests_compare/templates/comparison_report.html.j2 new file +414 -0

414 lines not on this page · raw message

scripts/generate_mmtests_graphs.py new file +598 -0

598 lines not on this page · raw message

workflows/mmtests/Makefile +14 -5
index 06a75ead..b65d256b 100644
@@ -13,19 +13,19 @@ mmtests:
 
 mmtests-baseline:
 	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
-		-l baseline playbooks/mmtests.yml
+		-l 'mmtests:&baseline' playbooks/mmtests.yml \
 		--extra-vars=@./extra_vars.yaml  \
 		--tags run_tests \
 		$(MMTESTS_ARGS)
 
 mmtests-dev:
 	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
-		-l dev playbooks/mmtests.yml \
+		-l 'mmtests:&dev' playbooks/mmtests.yml \
 		--extra-vars=@./extra_vars.yaml \
 		--tags run_tests \
 		$(MMTESTS_ARGS)
 
-mmtests-test: mmtests
+mmtests-tests: mmtests
 	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
 		playbooks/mmtests.yml \
 		--extra-vars=@./extra_vars.yaml \
@@ -39,6 +39,13 @@ mmtests-results:
 		--tags results \
 		$(MMTESTS_ARGS)
 
+mmtests-compare:
+	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
+		playbooks/mmtests-compare.yml \
+		--extra-vars=@./extra_vars.yaml \
+		--tags deps,compare \
+		$(MMTESTS_ARGS)
+
 mmtests-clean:
 	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
 		playbooks/mmtests.yml \
@@ -51,8 +58,9 @@ mmtests-help:
 	@echo "mmtests                       : Setup and install mmtests"
 	@echo "mmtests-baseline             : Setup mmtests with baseline configuration"
 	@echo "mmtests-dev                  : Setup mmtests with dev configuration"
-	@echo "mmtests-test                 : Run mmtests tests"
+	@echo "mmtests-tests                : Run mmtests tests"
 	@echo "mmtests-results              : Copy results from guests"
+	@echo "mmtests-compare              : Compare baseline and dev results (AB testing)"
 	@echo "mmtests-clean                : Clean up mmtests installation"
 	@echo ""
 
@@ -61,8 +69,9 @@ HELP_TARGETS += mmtests-help
 PHONY +: mmtests
 PHONY +: mmtests-baseline
 PHONY +: mmtests-dev
-PHONY +: mmtests-test
+PHONY +: mmtests-tests
 PHONY +: mmtests-results
+PHONY +: mmtests-compare
 PHONY +: mmtests-clean
 PHONY +: mmtests-help
 .PHONY: $(PHONY)
workflows/mmtests/fixes/0001-compare-Fix-undefined-array-reference-when-no-operat.patch new file +46 -0

46 lines not on this page · raw message

workflows/mmtests/fixes/0002-thpcompact-fix-library-order-in-gcc-command.patch new file +33 -0
new file mode 100644
index 00000000..0ab3e082
@@ -0,0 +1,33 @@
+From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
+From: Luis Chamberlain <[email protected]>
+Date: Tue, 5 Aug 2025 14:35:00 -0700
+Subject: [PATCH] thpcompact: fix library order in gcc command
+
+The gcc command in thpcompact-install has the libraries specified
+before the source file, which causes linking errors on modern systems:
+
+  undefined reference to `get_mempolicy'
+
+Fix by moving the libraries after the source file, as required by
+modern linkers that process dependencies in order.
+
+Signed-off-by: Luis Chamberlain <[email protected]>
+---
+ shellpack_src/src/thpcompact/thpcompact-install | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/shellpack_src/src/thpcompact/thpcompact-install b/shellpack_src/src/thpcompact/thpcompact-install
+index 1234567..7654321 100644
+--- a/shellpack_src/src/thpcompact/thpcompact-install
++++ b/shellpack_src/src/thpcompact/thpcompact-install
+@@ -8,7 +8,7 @@
+ install-depends libnuma-devel
+ 
+ mkdir $SHELLPACK_SOURCES/thpcompact-${VERSION}-installed
+-gcc -Wall -g -lpthread -lnuma $SHELLPACK_TEMP/thpcompact.c -o $SHELLPACK_SOURCES/thpcompact-${VERSION}-installed/thpcompact || \
++gcc -Wall -g $SHELLPACK_TEMP/thpcompact.c -lpthread -lnuma -o $SHELLPACK_SOURCES/thpcompact-${VERSION}-installed/thpcompact || \
+ 	die "Failed to build thpcompact"
+ 
+ echo thpcompact installed successfully
+-- 
+2.45.2
\ No newline at end of file
Signature · 1 line
2.47.2